DETERMINANT FORMULAS RELATING TO 
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Abstract. Chen et al. recently established bijections for (d+ l)-noncrossing/ 
nonnesting matchings, oscillating tableaux of bounded height d, and oscillat- 
ing lattice walks in the (i-dimensional Weyl chamber. Stanley asked what is 
the total number of such tableaux of length n and of any shape. We find a 
determinant formula for the exponential generating function. The same idea 
applies to prove Gessel's remarkable determinant formula for permutations 
with bounded length of increasing subsequences. We also give short algebraic 
derivations for some results of the reflection principle. 
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1. Introduction 

For a partition A = (Ai, . . . , A^)> of length (or height) at most d, we associate it 
with a A := X+(d, d — 1, . . . , 1). Then A belongs to the d-dimcnsional Weyl chamber 
defined by W d = {(xi,... ,Xd) : x\ > ■ ■ ■ > Xd > 0, Xi G Z}. In particular, we 
denote by = (d, d—1, . . . , 1) the associate of the empty partition 0. For A, p, G W d , 
let b n (X; p,) be the number of Weyl oscillating lattice walks of length n, from A to 
p, staying within W d , with steps positive or negative unit coordinate vectors. 

Theorem 1 (Grabincr-Magyar [8], Equation 26). For fixed A, p G W , we have a 
determinant formula for the exponential generating function: 

t n / \ 

flA A (*) = EMA;/i)^=det(j p4 _^(2t)-/p 4+Xj (2t)) (1) 

n>0 ~ 

where 

I s {2t) = [z*] exp(f(z + z- 1 )) = J2 u \ ^ 2n+S (2) 

is the hyperbolic Bessel function of the first kind of order s. 

Chen et al. [3] recently established bijections showing that (d + l)-noncrossing 
(nonnesting) matchings and oscillating tableaux are in bijection with certain Weyl 
oscillating lattice walks. Then Stanley asked (by private communication) the fol- 
lowing question: How many Weyl oscillating lattice walks of length n arc there if 
we start at but may end anywhere? Our main result answers this question: 

Theorem 2. The exponential generating function for the number of oscillating 
lattice walks in W d starting at = (d, d — 1, . . . , 1), and with no restriction on the 
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end points is 



G W := E E = det(J i _ i (2t)) 



l<ij<d, 



(3) 



n>0 tj.eW d 



where J s (2t) = [z s ] (1 + z) exp((z + z^t) = I s (2t) + I s -i{2t). 

Terminologies not presented here will be given in section 2, where we will explore 
the connection of oscillating tableaux with the Brauer algebra and symplectic group, 
just as that of standard Young tableaux (SYTs for short) with the symmetric group 
and general linear group. We will see that Theorem [2] actually gives a determinant 
formula for oscillating tableaux of bounded height, which is an analogy of Gessel's 
formula for SYTs of bounded height. 

Section 3 is for completeness of section 4, but is of some independent interest. 
We describe a simple algebraic derivation of the hook-length formula (Theorem [5]) 
and Theorem [T] as well as some notations. The method is easily seen to apply to 
many other results of the reflection principle. One can see from the proof a reason 
why using exponential generating function is preferable in this context. 

Section 4 includes the derivation of Theorem [2] Starting from the Grabiner- 
Magyar formula, one can obtain a constant term expression that can be used to 
do algebraic calculation. The theorem is then derived in three key steps: we first 
apply the Stanton-Stembridge trick (a kind of symmetrization), then a classical 
formula for symmetric functions, and finally reversely apply the Stanton-Stembridge 
trick. The same idea applies to prove a generalized form (Theorem [TTj) of Gessel's 
remarkable determinant formula [5]: 

Theorem 3 (Gessel). Let Ud(n) be the number of permutations on {1,2, ...,n} 
with longest increasing subsequences of length at most d. Then 



In this section, we will introduce many objects and try to explain their connec- 
tions with the classical objects for the symmetric group, in the view of enumeration. 
Some of the connections are in [T2l Section 9] , whose notations we shall closely fol- 
low. Finally, we will give some applications of Theorem [2] 

2.1. Notations. We assume basic knowledge of the symmetric group & n and its 
representation. See, e.g., [11] Chapter 7]. Now we introduce some objects. 

The Brauer algebra 25 n (depending on a parameter x which is irrelevant here) 
is a certain semisimplc algebra with the underlying space the linear span (say over 
C) of (complete) matchings on [2n] = {1,2,..., 2n}. The dimension of Q5„ is 



Its irreducible representations are indexed by partitions of n — 2r, for < r < [n/2\ . 
The dimension of the irreducible representation 25^ is equal to f£, that we are going 
to introduce. 

An oscillating tableau (or up-down tableau) of shape [i and length n is a sequence 
(0 = (j, 1 , . . . , fx n = (j.) of partitions such that for all 1 < i < n — 1, the diagram 





Our starting point is the well-known hook-length formula. 



2. Notations, connections, and applications 



dim<B„ = (2n- 1)!! = (2n - l)(2n - 3) • • • 3 ■ 1. 
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of [i l is obtained from by either adding or removing one square. Denote by 
f% the number of such tableaux. It is known that if \i is a partition of n — 2r for 
some nonnegative integer r, then 



where is the number of standard Young tableaux of shape fi. See, e.g., [2j 
Appendix B6] for further information. 

Denote by 9Jt„ the set of matchings on [2n] . A matching M € 9Jt n is a partition 
of [2n] into n two-blocks, written in the form {{i\,ji}, ■ ■ ■ , {i n , j n }}- We also write 
{ikijk) f° r {i-kijk} h < ife' We represent M by a diagram obtained by identifying 
i with [i, 0) in the plane for i = 1, . . . , 2n, and drawing arcs, called edges, from 
to jfc above the horizontal x-axis for all k. For d > 2, a d-crossing of a matching 
M is a set of d arcs (iruiri)) (v 2 ;iT- 2 )) • • • j (ird)jr d ) °f such that i ri < i r2 < 
• • • < i rd < j ri < > 2 < • • • < jr d ■ A matching without any d-crossing is called a 
d-noncrossing matching. We omit here the similar definition of c?-nesting. Figure [T] 
shows the diagram corresponding to the matching 



Figure 1. A matching on [10], in which the edges 
{1, 4}, {2, 8}, {3, 10} form a 3-crossing. 

Now we introduce apparently new objects. For an oscillating tableau O of shape 
(hence of even length), reading O backwardly still gives an oscillating tableau 
of shape 0, denoted by O rev . We say that O is palindromic if O = O rev . For a 
matching M of [2n] , let M re f l denote the matching obtained from M by reflecting 
in the vertical line x — n + 1/2. Figure [2] shows the diagram corresponding to 
M re f l . Then M is said to be bilaterally symmetric if M = M re f l . Equivalently, 
(i, j) is an edge of M if and only if so is (2n + 1 — j, 2n + 1 — i). 





M = {{1, 4}, {2, 8}, {3, 10}, {5, 7}, {6, 9}}. 




123456789 10 




123456789 10 



Figure 2. The reflection of the matching in Figure [T] 
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Proposition 4. The exponential generating function of the number bsm„ of bilat- 
erally symmetric matchings on [2n] is 

fn 

> bsnin— - = exp(i + t 2 ). 
z — ' n! 

n>0 

Proof. For a bilaterally symmetric matching M on [2n] , identify it with the graph 
M' obtained from M by adding the (dashed) edges (i, 2n+ 1 — i) for i = 1, 2, . . . , n. 
Then every vertex of M' has degree 2, so that M' can be uniquely decomposed into 
connected components, each being a cycle. The cycles can be of only three types, 
as drawn in Figure [3J Therefore, the lemma follows by the well-known exponential 



Figure 3. The types of connected components of bilaterally sym- 
metric matchings. 

formula for generating functions. See, e.g., [TTJ Corollary 5.1.6]. □ 
2.2. Connections. We first give a list of the classical objects and their analogies: 



Classical Objects 



the symmetric group &„ 
the general linear group GL(d) 
standard Young tableaux 
permutations on [n] 
involutions 



Their analogies 



the Brauer algebra Q3„ 
the symplectic group Sp(2d) 
oscillating tableaux 
matchings on [2n] 
bilaterally symmetric matchings 

Next we give connections in the view of enumeration. By a well-known result in 
representation theory, we have 

J2 (/£) 2 = (2n-l)H, £(.T) 2 = n!, (5) 

fih(n— 2r) flhn 

where the first sum ranges over all nonnegative integers r with < r < \n/2\ 
and partitions fi of n — 2r. The former equation of ^ is for 25„ and the latter 
one is for 6 n ; f% and / M are the dimension of the corresponding irreducible rep- 
resentations. We shall always list the analogous formula before the classical one. 
Equation (O suggests a RSK-correspondence for matchings just as that for permu- 
tations. Observe that a pair of oscillating tableaux of the same shape of length n 
can be naturally combined as one oscillating tableau of shape of length 2n. To 
be precise, the decomposition 7(0) = (P, Q) is given by 

7 : (0 = /A (i\ . . . , \i 2n = 0) — > ((0 = fjP, (i 1 ,..., (0 = /X 2 ", M 2 "" 1 , . . . , /i")). 

Thus it is sufficient to construct a bijection from the set 97l„ of matchings to 
the set O n of oscillating tableaux of the empty shape and length 2n. Such a 
bijection was first given by Stanley (unpublished), and was extended by Sundaram 
[13] to arbitrary shapes to give a combinatorial proof of the Cauchy identity for 
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the symplectic group Sp(2d), and was recently extended by Chen et al. [3] for 
partitions. 

Let $ be the bijection from DJl n to O n defined in (T2J Section 9]. Then it was 
shown in [3] that <& has many properties. We will use the fact that the maxi- 
mum number of crossings of a matching M is equal to the maximum height of the 
oscillating tableau $(M). 

We will use the following result of [15]: For any M e 97t„, we have $(M re -^) = 
<f>(M) rev . 

Since the number of palindromic oscillating tableaux is equal to the number of 
bilaterally symmetric matchings bsm„, we have, by Proposition |4l 



E 

li\-(n-2r) 



J a 



exp(i + i 2 ), 



fihn 



exp(i + t 2 /2), 



(6) 



where the first sum ranges over all nonncgative integers r with < r < \n/2\ and 
partitions \i of n — 2r. The right equation of §§§ counts the number of involutions. 

There are analogous results if we put restrictions on the height of the tableaux. 
A d-oscillating tableau, also called d- symplectic up-down tableau, is an oscillating 
tableau of a bounded height d, by which we mean that the height of every /j, 1 is no 
larger than d. Denote by f%(d) the number of d-oscillating tableaux of shape pi and 
length n. See [13] for more information. 

There is a natural bijection showing that f£(d) = b n (0, p), which has a determi- 
nant formula as in ((TJ. The bijection simply takes p 1 , . . . , /i") to the sequence of 
lattice points (pP, p, 1 , . . . , p n ). Therefore results on oscillating lattice walks can be 
translated into those on oscillating tableaux. Applying 7 to rf-oscillating tableaux 
of shape and length 2n, and applying Theorem [1] we obtain 

i2n 



E 

fj,h(n— 2r) 



(fH(d)Y = b 2n (o,o) 



t 1 



(2n)! 



det (2i) -I i+j {2t)) 1 



<i,j<d ■ 



(7) 



an analogy of Theorem[3] This is also the number of (e?+l)-noncrossing/nonncsting 
matchings. See [3l Equation (9)] (by setting k = d+ 1) and references therein. 
Theorem [2] actually gives 



E 

iul-(n-2r) 



t" 



det(Ii-j(2t) + 7 i _ 3 -_i(2t))i<i J < d . 



This is an analogy of Gessel's determinant formula for involutions. See [5]. See also 
QJ Sections 4&5]. 

Let is(w) be the length of the longest increasing subsequences of w £ & n , and let 
cr(Af) be the maximum crossing number of M £ Then we have the following 
table. 



Classical Objects 


Their analogy 


the general linear group GL(d) 
SYT of bounded height d 
{w € &„ : is(w) < d} 

& is an involution 


the symplectic group Sp(2d) 
oscillating tableaux of bounded height d 
{M e Tl n ■■ cr(M) < d} 

& is bilaterally symmetric 



Stanley |12j gave a nice survey for the study of increasing and decreasing subse- 
quences of permutations and their variants. One major problem in this area is to 
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understand the behavior of is(u>). For instance, what is the limiting distribution 
of is(w) for permutations? Gessel's determinant formula reduces such problem to 
analysis, which was solved by Baik, Dcift, and Johansson [T] using their techniques. 
In our table, the limiting distribution formulas of is(w) for permutations and for 
involutions, and that of cr(Af) for matchings are known. The distribution for bi- 
laterally symmetric matchings should be obtained in a similar way, but this needs 
to be checked. 

2.3. Applications. We first summarize several consequences of Theorem O 
Corollary 5. The following quantities are equal to 

det(Ji_j(2t) + / i -j_i(2t))i< iJ < d . 



(1) The number of palindromic Weyl oscillating lattice walks of length 2n and 
starting at 0. 

(2) The number of palindromic oscillating tableaux of length 2n. 

(3) The numberhsYn n (d) of bilaterally symmetric (d+1) -noncrossing /nonnesting 
matchings on [2n] . 

(4) The number of oscillating tableaux of any shape and length n. 

We can compute bsm„(d) for small d. For the case d = 1, we have 

. , / 2n\ , , (1 l/2ii + 2\ 

bsm 2 „(l) = , and bsm 2n+ i(l) = - . (8) 

V n J 2 \ n + 1 J 

This is a direct consequence of Theorem [21 but we give an alternative proof. 

Proof of ©. By Corollary O part (3), we need to compute noncrossing bilater- 
ally symmetric matchings on [2n\. Let P(t) be the generating function P(t) = 
^2n>o bsm„(l)t". Consider the possibility of the edge (1, to) in the bilaterally sym- 
metric matching M. One sees that if to > n, then to must equal 2n to avoid a 
crossing. Thus we have the decomposition of M as in Figure [H where we use semi- 
circles to indicate noncrossing matchings, and use trapezoid to indicate noncrossing 
bilaterally symmetric matchings. Therefore, we obtain the functional equation: 




Figure 4. Decomposition of noncrossing bilaterally symmetric matchings. 

P(t) = l + t,P(t)+t 2 C(t 2 )P(t), 

where C(t) = 1 ~^~ 4t is the Catalan generating function, which is known to be 
the ordinary generating function for noncrossing matchings. Direct algebraic cal- 
culation shows ([5]). □ 
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For the case d = 2, we have 

l/2n + 2\ (2n+l)!(2n)! 
bsm 2n (2) = 2^ n + 1 JCW= (n!(n + 1)!)2 . 0) 

/N l/2rc + 2\ (2n + l)!(2n + 2)! , , 

bsm 2n+1 2 = - C n+1 = \ / ' 10 

2 \ n + 1 / n!(n + l)! 2 (n + 2)! 

For the case d = 3, we obtain 

i 2(2a + 1)1 (271)! 

bsm 2 „ [6) - ^ s[2(s - - 2) , • ^ - g)l{n _ - - [U) 

i A 2(2s + 2)! (2n+l)! 

bsm 2 „ + i(3 => - ? — — — . 12 

s!(s + l)!(s + 2y. z [n — s)l[n — s + 1)! 

By a general theory, these sequences are P-recursive, or their generating functions 
(for any d) are D-finite. See, e.g., [TTJ Chapter 6]. We use the creative telescoping 
of [9] to find that bsm 2n (3) satisfies a second order P-recursion [TTJ Chapter 6]: 

(n + 5) (n + 4) (n + 3) bsm 2n+4 (3) 

= 4 (5n 2 + 30n + 43) (2n + 3) bsm 2 „ +2 (3)-36 (2n + 3) (2n + 1) (n + 1) bsm 2 „(3), 

(13) 

subject to bsm (3) = l,bsm 2 (3) = 3; and that bsm 2 „ +1 (3) satisfies a P-recursion 
of order 3, which is too lengthy to be given here. 

Formulas (|9ll0p are straightforward by, e.g., the creative telescoping. Formulas 
(|11I12[) need some work. We will write Ii for Ii{2t) for short, and use the following 
facts: /, = I—i, In contains only even powers in t, and hi+i contains only odd 
powers in t. 

Proof of UH12\) . By Theorem [21 the exponential generating function is 

/ h+h h + h h + h 
det h+h h+ h h + h 
\ h+h h + h h + h 

= (h - I 2 ){{h 2 - h 2 - I 2 2 + hh) + (hh + hh - 2Ji/ 2 )), 

where h — h contains only even powers in i, and in the right factor, we have 
separated the sum according to the parity of the powers in t. 

Now it is straightforward, by the creative telescoping, to show that 

t 2n 



»!(n + l)!' 
2(2n+ iy.t 2 



n>0 

i2n 



h 2 - I \ - h 2 + hh — 2_\ 

^n\\n+l)\(n + 2)\ 

2{2n + 2)\t 2n+1 
n\(n + l)\(n + 2)\ 2 ' 

n>0 y ' V ' 



hh + hh - ^hh = 



Equations Q11I12|) then follow. □ 

We remark that {bsm 2 „(2)} ra >o gives the sequence A000891 in the Online En- 
cyclopedia of Integer Sequences [TU]. One of its interpretation can be stated in 
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our term: it counts the number of noncrossing partitions of [2n + 1] into n + 1 
blocks. We also remark that {bsm2 ra (3)} n >o gives the sequence A064037 in [TO] . 
The only known interpretation is: it counts the number of 3-dimcnsional oscillating 
lattice walks of length 2n, starting and ending at the origin, and staying within the 
nonnegative octant. Bijections for these objects are desirable. 

3. Algebraic Description of the Reflection Principle 

A classical application of the reflection principle is to the ballot problem, which, 
in random walks version, asks how many ways there are to walk from the origin to 
a point (Ai, . . . , Xd)>, with each step a positive unit coordinate vector and confined 
in the region x\ > x-i > ■ ■ ■ > Xd > 0. The reflection principle of |H [16] gives a 
determinant formula, from which the hook-length formula for SYTs can be deduced. 
Our objective in this section is to give short algebraic derivations of this formula 
and the formula of Grabiner and Magyar. 

In the context of lattice walks, it is convenient to shift the coordinates a little 
and denote by W d = { (xi, . . . , Xd) ■ x\ > ■ ■ ■ > Xd > 0, x t G Z } the rf-dimensional 
Weyl chamber. From now on, A, fj, will not denote partitions as in previous sections. 
Let 8 = (d, d— 1, . . . , 1). Then any /j, £ W d corresponds to a unique partition /i — 5. 

3.1. Hook-Length Formula. 

Theorem 6 (Hook-Length Formula). The number of standard Young tableaux of 
shape A is 

f ^ {Xl + ... + ^_J_^^. (14) 

Fixing a starting point A e W , we let /(A; /x) be the number of W -walks from 
A to /i, with only positive unit coordinate vector steps. Clearly, the length of such 
walks, if exist, is |^| — |A|, where \(i>\ = (j>i + • • • + fid- Then the number f^~ s of 
SYTs of shape /i — 5 equals f(5 7 fi). 

Let F(x) = F(xi, . . . , Xd) be the generating function 

F(x u ...,x d ) = /( a ;m)^, 

fj,GW d 

where x^ = x^x!^ 2 ■ ■ -x^ d records the end points. From known results, F(x) is 
D-finite and does not have a simple expression. But F(x) has a simple rational 
function extension: 

m - j , r X(X l T y (15) 
1 - (Xl + X2 + h Xd) 

where a\(x) is the alternant det ( x^ 3 ] 

V J l<i,j<d 

Proposition 7. Let F(x) be as above. If we expand 

p & = i 1 T^Z z \ = E fto*w> 

1 - (a;i + ica H \-x d ) ^ 

then /(A; /i) = /(A; /j.) for all /z in the closure of W d . 

Proof. Let e 2 ; be the ith unit coordinate vector. Let x(S) = 1 if the statement 5* 
is true and otherwise. Then for \i in the closure of W d , /(A; fi) can be uniquely 
characterized by the following recursion: 



DETERMINANT FORMULAS RELATING TO TABLEAUX OF BOUNDED HEIGHT 9 



(i) If H<|A|,then/(A;M)=x(M=A). 

(ii) If fii = iii+i for 1 < i < d — 1, then /(A; fj) = 0. 

(iii) If \n\ - |A| > 0, then /(A; fi) = £ti /(A; /i - e*). 

Therefore, it suffices to show that /(A; //) also satisfies the above three conditions. 
Condition (iii) is trivial according to (fT5)k Condition (ii) follows easily from 

F(xi, . . . , aci_i, Xi+i, Xi, x i+2 , . . . , x d ) = -F(xi, ...,x d ); 

To show condition (i), we notice that the numerator of F(x) is homogeneous of 
degree |A|. and the least degree term in the series expansion of (1 — Xi — ■ ■ ■ — .r,/) _1 
is 1. This implies that if |?7| < |A|, then /(A; 77) equals (— l) 7r if r\ — 7r(A) for some 
7r £ &d and zero otherwise, where (— I) 7 ' is the sign of tt and 7r(A) = (A^j, . . . , A ffd ). 
Condition (i) follows since 7r(A) £ W only if 7r is the identity. 

This completes the proof. □ 

By setting A = S, one can derive the hook-length formula, Theorem [6] This 
completes our first objective of this section. 

3.2. Grabiner-Magyar Determinant Formula. The same argument applies to 
more general situations, such as with a different set of allowing steps. We give one 
more example to illustrate the idea. Note that the underlying idea is the reflection 
principle. 

Fix a starting point A £ W d (the most interesting case is A = 8). Let & n (A;/x) 
be the number of Weyl oscillating lattice walks of length n from A to fi. Note that 
we changed the notation here. The A is abbreviated by A, and similar for p,. 

Proposition 8. For fixed A £ W d , let 

B x (x;t) = &^-x- X ^,< d (16) 

1 - t[x\ + x 1 + x 2 + x 2 + • • • + Xd + x d ) 

Then [x^t™] B\(x; t) — b n (X; fi) for any /x in the closure of W d and n£N. 

Proof. Clearly 6„(A; jj,) is uniquely determined by the following recursion: 

(i) If n = then 6„(A; fj,) = xip- = 

(ii) If fii = Hi+i for 1 < i < d — 1, or if fid = 0, then 6 n (A; /x) = 0. 

(iii) If n > 1, then b n (X; n) = £\ =1 &„_i(A;/i- e t ) + 6„_i(A; fi + e,). 

Denote by b n (X;n) = [x v t n ] B\(x; t). It suffices to show that 6 n (A;/x) satisfies 
the same recursion as for b„ (X; fi) when fi belongs to the closure of W d . Con- 
dition (i) is straightforward; Condition (ii) follows from the identities B\(x;t) = 
- B x (x;t)\ m+uXi+i=Xi for 1 < i < d - 1, and B x (x;t) = - B x (x;t)\ Xd= _ Xd ; 
Condition (iii) follows by writing 

B x (x; t)(l - t(xi + x^ 1 + x 2 + x^ 1 H h x d + x^ 1 )) = dct(x^ - x i A3 )i< l . J <d 

and then equating coefficients. □ 

Now we are ready to complete our second task of this section. 



Proof of Theorem [II By Proposition [HJ it remains to extract the coefficient of .t' 1 
in Bx(x;t). 
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Since the numerator of B\(x;t) is independent of t, it is easy to obtain the 
exponential generating function 

^ b n (X; rj)x v — = det(x t 3 - x l 3 )i<i,j<d exp(i(xi + xT/ 1 H hx^ + x^ 1 )) 

n>0nez d 



det 



((x? -x7 x >)exp(t(x i + x; 1 ))) i 



<i,j<d 



Now taking the coefficients of x^ 1 ■ ■ ■ x^ d yields 

r = det | 1x7" J I exp(i(x, ; + x, ")) - |x7" ' J I exp(i(x, ; + x, ■ ")) ] 

l<i,j<d 



V 6„(A; n)-= det f[xf exp(i(x, + x" 1 )) - [xf +Aj ] exp(t(x, + x" 1 ))) 
z — ' n! V / 1< 



n>0 



which is equivalent to (TT]). □ 

Note that we use exponential generating functions because exp(i(xi +X7/ 1 + • • ■ + 
Xd + x^ )) factors nicely enough to be put inside the determinant. 



4. Two Formulas Relating to Tableaux of Bounded Height 

In this section, we will prove Theorems [2] and [31 where the former is a new result 
and the latter is Gessel's remarkable determinant formula. We will first express 
our objects as certain constant terms. Then we will play two tricks, the Stanton- 
Stembridge trick and the reverse of the Stanton-Stembridgc trick, in evaluating 
such constant terms. 

4.1. Stanton-Stembridge Trick. Fix a working ring K. that includes the ring 
C((xi, . . . , Xd)) of formal Laurent series as a subring. For example, in our applica- 
tions, the working ring is K = C((xi, . . . , [[£]]. A permutation ir £ &d acts on 
elements of K, by permuting the x's, or more precisely 

n ' a ii,—,id x 'l'" x 'd = a ii,---,i d x -iTi ' ' ' x ir d - 

ii,...,id& i\,...,id& 

We say that K. is &d- invariant if n ■ 1C = K, for any ix G &d- For example, the 
ring K, = C((xi, . . . , Xd))[[i]] is G^-invariant, but the field of iterated Laurent series 
C((xi))((x2)) is not 62-invariant (see [14]V 

In what follows, we always assume that K. is ©^-invariant. One can easily check 
that this condition holds in our application. 

Lemma 9 (Stanton-Stembridgc trick). For any H(x±, . . . ,Xd) £ /C, we have 
CT H{x 1 ,...,x d ) = —. CT V] tt ■ H(x 1 , . . . ,x d ), 

Xi,...,X d d\ Xl,...,X4 * ' 

iv£6 d 

where CT X i,...,x d means to take the constant term in the x's. 

The lemma obviously holds. The following direct consequence is useful. 

Corollary 10. Suppose that H,U,V £ K, and that U(x) = U(xi,...,Xd) and 
V(x) = V(x±, . . . , Xd) are symmetric and antisymmetric in the x's, respectively. 



Then 
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CT H{x u ...,x d )U(x) = ^ CT U(x) V n ■ H(x u . . . ,x d ), 
c 1 ,...,x d a\ xi,...,x d *■ — ' 

CT H(x u ...,x d )V(x) = ±- CT V(x) V {-l^n ■ H(x u . . . ,x d ). 
ci,...,x d d\ xi,...,x d *■ — ; 



Tree d 



We call both the lemma and the corollary the Stanton-Stcmbridgc trick (SS-trick 
for short). See, e.g., [T7J, p. 9]. 

4.2. Proof of Theorem [2j First let us write gs^{t) a s a constant term using 
Theorem [T]and the fact that I s {2t) = I- S (2t). 

g hl (t) = dct (CT [xf -*> exp((x; + X T x )t) - x^ +S] exp((x l + x^t)] ) 

\ Xi 1 V l<i,j<d 

By factoring out x^ 1 exp((:Ei + x" 1 )^) from the ith row, we obtain 

d 

g-^(t) = CTdet (x~^ ~ x*A J[ x? cxp((x l + x^)t). 

i=l 

Therefore, G(t) can be expressed as a constant term in the x's: 

d 

G(t)= J2 = CTexp(^(x ! + a; - 1 )t)det( 2 ;- 5j - xf) ■ £ a* 

peiv d »=l ^GW d 

Now we can apply the SS-trick to obtain 
1 d 

G(t) = - CTexp (a* + xi 1 ) t) det (x^ - xf ) £ ]T (-1)** • x", 

(17) 

where we used the fact that the first factor is symmetric and the second factor is 
antisymmetric in the x's. 

The determinant is well-known to be equal to 



d 

det (xT s > - x 5 A = - xf ) (1 - x^a^x" 1 ), (18) 



l<i<j<d 

_-l -1- 



where a; 1 = (x 1 , . . . , x d 1 ), and the alternants are related to the Schur functions 
as follows: 

By the above formula, and the classical identity [ITJ Equation 7.52] for symmetric 
functions (by setting Xk = for k > d), we obtain 

]T J2 i-i)**-** 1 =«»?(*)— ■ (19) 

(igi^i 7rGe d rii=i(i _ rii<i<j<d(i _ xjXj) 
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Now substitute (fT8|) and (fT9jl into (fl"7]) . After a lot of cancelations, we obtain: 

^ d d 

G(0 = ^CTexp(^(^+^ 1 )i)n( 1 + ^)^( a; " 1 ) E • ( 2 °) 

i=l 2=1 7rG©d 

Now aj(a; _1 ) is antisymmetric. Reversely applying the SS-trick to (|20|) gives 
G(t) = CTx? • • ■ x d a- 5 {x- x ) Y[ exp((x 4 + a;" 1 )*) + a*) 

i=l i=l 
d d 

= CTdet (xf 1 ) TTexp((x l + xT^t) TT(1 + x t ) 

Z— 1 Z— 1 



= det ^CTxf _i (l + exp((x 8 + xT l )tyj 
which is easily seen to be equivalent to ©. 



<i,j<d 



4.3. Gessel's Determinant Formula. The tricks for proving Theorem[3]are sim- 
ilar as in the previous subsection. We remark that previous proofs of this result 
rely on the powerful tools of symmetric functions. See, e.g., [5J \7\. 
It follows from the RSK-corrcspondence that 

u d (n)= rr= E /(M 2 , 

\a\=n \n\=n+\5\,fieW d 

where a ranges over partitions of n of height at most d. We will find a generating 
function of 

u d (\;n) := ^ /(A;/i) 2 . 

More precisely, we have the following generalized form. 
Theorem 11. Let A e W d and let I s (2t) be as in Theorem^ We have 

t 2n 

U d (X;t) = Y^UdiX-n)— = det(/ Ai _A 3 )i<i,i<rf- (21) 

n>0 

Note that /(A;/i) is the number of standard skew Young tableaux of shape 
(/i — S)/(X — 5). See [lT| Equation 7.7.1] (note that there is a change of indices). 
Therefore u d (X; n) counts the number of pairs of standard skew Young tableaux of 
the same shape (fj, — S)/(X — 5) with \fi\ — \X\ = n. 

Proof of Theorem \lll By Proposition 

/(A; n) = [x") a x (x) ^(xi + x 2 + ■ ■ ■ + x d ) k . 

fc>0 

Note that when taking the coefficient in x M , only the summand with respect to 
k = n has a contribution, where n = — |A| is the length of the lattice walks. It 
follows that 

t n 

/(A; fi)—r = [x M ] a\(x) exp ((xi + x 2 H h x d )t) 



= det([xf _A3 ]exp(tXi)) 



l<i,j<d 
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When written in constant term, we obtain 
t n 



f(X;n)— = CTdet(cc i 3 )i< i . :) < d TTa; l Mi exp(ixi) 

»=i 

= CT a\{x~ l )xi 1 ■ ■ -xtf exp(i(xr 1 + ■ • • + x" 1 )), (22) 



where the last equality follows by substituting x i 1 for xi. 

Now squaring both sides of (|22[) and summing over all [i^ we obtain 



j.2n 

U d (X;t)= £ /(A;/i) 2 ^ 



a a 

CTa^x-^^cxp^x- 1 ) •CTa A ( 2 /- 1 )^cxp(t^ 2/ ,- 1 

fiGW d J »=1 i=l 



d d 

= CTa A (x- 1 K(y- 1 )exp(^x- 1 +^ 2 /- 1 ) £ x^. (23) 

j=i »=i (lew 

We need the following easy formula: 

^ (-ir(7r. a *)(-l)>.^) = o M (x)o„(»), (24) 

7T,crGSd 

and the well-known Cauchy-Binnet formula (see, e.g., [TTJ p. 397]): 

a M (x)a M (y) = x V det ( — ) , (25) 

V 1 x iVo J l<i,j<d 

where 1 is the vector of d l's and x 1 = X1X2 • • • aid- 
Now apply the SS-trick to (|23|) for the x- variables and the y- variables separately, 
and then apply (]24[). We obtain 



d d 

C/d(A;t) = -pCTa A (x- 1 )aA(2/" 1 )exp(tJ]x~ 1 +^y- 1 ) ^ a /1 (x)a /1 (y) 



(i! 2 

Applying ([23]) gives 

i ^ /i 

W;t) - ^ I CTa A (x- 1 )a A (y- 1 )exp(i^(x- 1 + 2/ r 1 )). T i y i det I — 



Clearly, the last determinant is antisymmetric in the x-variables and also in the 
y-variables. Reversely applying the SS-trick for the x's and for the y's, we obtain 

U d {\-t) = CTx-V A cxp (tJ2 (x- 1 +J/- 1 ) )x 1 y 1 dct ( —± ) 

x >y v s 1 V 1 - x iVj J i<i,3<d 

= CTdet fx] Ai y] A ' exp^x" 1 + tyj 1 )—^ ) 

V 1 - x iVj J l<i,j<d 

= det ( CT x] V. A ' cxp(txr 1 + i^ 1 )-^ ) . (26) 

\ X «'W J 3 1 ~ x iVj J l<i,j<d 
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We finally need to evaluate the entries (constant terms) of the above determinant. 

J k,l>0 m>0 

V * - . 

^ (1 - \i + m)\(l - \j + m)\ 

By changing the indices 1 — A.; + m = n, we obtain 

l-Ai l-Aj 

CT cxp^a-- 1 + tyj 1 )^--^ = I Xi _ Xj (2t). (27) 

This completes the proof. □ 

By going over the proof, one can see that the a\(y) may be replaced with a u (y) 
without making much difference. This gives the following proposition. 
Let A, v e W d with |A| > \v\. Denote by 

E t 2n+\\\-\u\ 
Ud{^v\n)— — —— — 
„> n\(n+\\\ -\v\)\ 

where 

u d (\;v;n) := ^ f(\;fi)f(v;n). 

\ft\=n+\\\,p,EW 

Proposition 12. For X,v eW d with |A| > | v\, we have 

U d (\;v\t) = det(I Xl - U] )i<ij<d- 

Gcsscl's determinant formula was proved by first deriving a symmetric function 
identity, and then applying a specialization operator. It is not a surprise that there 
should be a corresponding symmetric function identity that specializes to Theorem 
[TT1 even to Proposition [12] Actually, such a formula was described by Gessel in the 
same paper [5] in the paragraph just before Theorem 16, and was clearly stated as 
[1 Theorem 3.5]. 

Gessel's determinant formula counts permutations of bounded length of longest 
increasing subsequences. Does Theorem [TT1 count natural objects? 
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